In the video and film industry, the animation process can take any number of various forms: conventional cel animation, the use of three dimensional (xe2x80x9c3-Dxe2x80x9d) models and effects and image-based rendering. Cel animation is the traditional method for generating animation by painting a foreground object on a sheet of clear celluloid (cel) and placing the cel on a background. The animation is generated by producing a series of cels with the foreground object moved to a new position in each cel. A 3-D model of an object is a representation that contains geometry information and surface attributes for a 3-D object in order to display the 3-D object on a 2-D display. The geometry information in the representation typically defines the surfaces of the 3-D object as a list of flat polygons sharing common sides and vertices. The surface attributes in the representation specify the texture and color to apply to the 3-D object. Both the generation of cels for cel animation and the generation of 3-D models are laborious and time consuming.
Image-based rendering techniques use 2-D images for visualizing in 3-D as well as editing and manipulating 3-D static scenes. The 3-D effects are generated through a rendering of a novel view of the 3-D static scene from another viewpoint by moving a virtual camera position. An image-based rendering technique for generating 3-D effects from a set of two-dimensional (xe2x80x9c2-Dxe2x80x9d) images of a 3-D static scene is described in Greene et al. xe2x80x9cCreating raster Omnimax images from multiple perspective views using the Elliptical Weighted Average filterxe2x80x9d, IEEE Computer Graphics and Applications. pages 21-27, August 1995. An image based rendering technique such as described in Greene et al. requires a set of 2-D images from multiple perspective views of the 3-D static scene. The set of 2-D images is required to provide the 3-D geometric information to render novel views of the 3-D static scene.
Current image-based rendering techniques such as the technique described in Greene et al. may be used only when there is a set of 2-D images of the 3-D static scene available, to provide 3-D geometric information. However, there may be only a single 2-D image of a 3-D static scene, for example, a painting or a photograph or the single 2-D image may be an abstract painting with no depth. A technique to generate 3-D effects for a single 2-D image of a 3-D static scene is described in xe2x80x9cTour into the picture: Using a spidery mesh interface to make animation from a single imagexe2x80x9d, by Horry et al. in Computer Graphics (SIGGRAPH""95), pages 229-238, August 1995. Horry et al. provides a user interface which allows the user to create 3-D effects from a single 2-D image. After the user has selected a vanishing point and distinguished foreground objects from background objects in the single 2-D image, a 3-D model is generated through the use of simple polygons and planar regions. Novel views of the single 2-D image are generated using the generated 3-D model of the single 2-D image.
An image taken from a new point of view may be rendered from an original two-dimensional image without a 3-D model using the present invention. Multiple-layers are defined by assigning depth to pixels in the original image, and pixels with depth are added to portions of layers covered by a foreground layer in the original image. Image locations of pixels from the original image and added pixels, which are reprojected to a new point of view, are calculated from the new point of view and dimensional information of the original image, including the assigned depth. The reprojected pixels are displayed at the calculated locations.
On a preferred and novel user interface, the original two-dimensional image and images from new points of view are simultaneously displayed during the rendering process. The original two-dimensional image may be centered within the images calculated for points of view left, right, up and down relative to the point of view of the original two-dimensional image.
Depth may be assigned to pixels in the two dimensional original image by designating regions and assigning depth to pixels within each region as a class. The depths of the pixels within a region may be assigned as a function of pixel position within the region. For example, the depth of a pixel may be proportional to its distance from a region boundary. The depths of pixels within a region may also be assigned as a function of pixel brightness in the region.
Depth may be assigned to a region such that the region is rotated relative to the original image. The resultant gaps between boundaries as viewed from new points of view may be filled by recomputing depths of pixels in adjacent regions to create smooth depth transitions between the region boundaries.